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/ Abstract 

'“"he  Army  Pesearch  Institute  for  the  Behavioral  and  Social  Sciences  (API)  is  developing  a 
data  base  on  the  relationship  between  training  simulator  charateristies  - in  particular  fidelity 
- and  training  effectiveness.  Tn  order  to  guide  and  organize  the  collection  of  empirical  data 
for  the  data  base,  a two  factor  definition  of  simulator  fidelity  was  chosen.  The  definition  was 
limited  to  physical  and  functional  slmilaritv  to  the  actual  eouipment.  Several  research 
efforts,  using  this  definition,  are  currently  being  conducted  or  have  Just  reached  completion. 
_ This  paper  presents  the  results,  to  date,  of  completed  research  efforts,  the  anticipated  results 
^ 3 of  ongoing  efforts,  and  plans  for  future  efforts.  The  goal  of  these  efforts  is  to  produce  a 
data  base  which  mav  serve  as  the  foundation  for  the  development  of  systematic  guidance  to 
support  the  specification  of  training  device  characteristics.. 

Introduction  ,\  , , 


T nt  rocluc  t i on 

The  Army,  the  other  services  and  the 
training  community  lack  specific  guidance  to 
determine  the  characteristics  which  will  enable 
training  simulators  to  provide  effective 
training  at.  affordable  costs.  In  the  past  it 
has  too  often  been  the  case  that  the  character- 
istics of  a training  device  or  simulator  were 
determined  bv  a mix  of  intuition  (what  someone 
thinks  the  device  should  be  like)  and  cost  (buy 
as  much  "realism"  as  we  can  afford).  This 
approach  to  training  simulator  design  is  no 
longer  a viable  option  given  budget  and  other 
resource  constraints.  Accordingly  ART  has 
undertaken  a program  of  research  to  develop  user 
oriented,  empirically  based  guidance  to  aid  in 
determining  the  characteristics  of  training 
simulators  which  will  maximize  training  effec- 
tiveness. The  program,  from  its  conceptual 
development  stage,  through  two  ongoing  empirical 
efforts  and  finally  to  its  future  goals,  is 
summarized  in  this  paper. 

Defining  Training  Simulator  Fidelity 

The  first  step  in  developing  API's  training 
simulator  research  program  was  to  adopt  a 
working  definition  of  simulator  fidelity.  An 
extensive  literature  search'  ' indicated  a large 
amount  of  inconsistency  in  how  fidelity  has  been 
defined  bv  the  training  RAn  community.  Tn 
particular,  the  term  has  been  used  somewhat 
lndiscriminant.ly  to  label  different  categories 
of  Independent,  and  depent  variables  which 
characterize  both  training  systems  and 
trainees.  It  was  concluded  that  a limited, 
parsimonious  definition  or  fidelity  would  best 
serve  the  compilation  of  a data  base  on  how 
device  characteristics  are  related  to  training 
effectiveness.  Accordingly,  the  following 
definition  was  chosen: 


1)  The  physical  characteristics  of  the 
training  simulator. 

?)  The  functional  characteristics 
(i.e.,  the  informational  or  stimulus  and 
response  options)  of  the  simulated  eouipment. 

Tt  was  further  determined  that  for  purposes  of 
empirically  testing  and  validating  this 
definition  as  well  as  for  generating  empirical 
data  on  the  relationship  of  training  simulator 
fidelity  to  training  effectiveness,  a three 
level  ordinal  scale  would  be  used  for  each 
aspect  of  fidelity  (physical  and  functional). 
This  approach  yields  the  matrix  like  the  one  in 
Figure  1.  This  nine  cell  matrix  serves  as  the 
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Training  simulator  Fidelity  is  the  degree 
of  similarity  between  the  training 
simulator  and  the  eouipment  which  is 
simulated.  It  is  a two  dimensional 
measurement  of  this  similarity  in  terms  of: 


Flgura  1:  Nine  call  Matrix  for  Training  Simulator 

Fidelity  Experiments 

"basic"  experimental  design  for  use  in  sub- 
seouent  empirical  efforts. 

Descriptions  of  possible  devices  which 
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would  fit  Into  each  cell  of  Figure  1 may  clarify 
this  research  approach.  Cell  HH:  The  actual 

equipment  could  serve  as  the  exemplar  of  a high 
physical-high  functional  device  and  would  there- 
fore fit  into  the  high-high  cell  of  Figure  1. 

Cell  HH:  The  high  functional-medium  physical 

cell  could  contain  a fully  functional  device  but 
one  with  some  combination  of  reduced  size,  number 
and  accuracy  of  its  components  relative  to  the 
actual  equipment.  Cell  HL:  The  high  functional- 

low  physical  cell  could  contain  a computer  graph- 
ics type  device.  This  device  would  consist  of 
Interactive  computer  graphics  which  would  func- 
tion In  an  analagous  manner  to  the  actual  equip- 
ment but  which  would  look  like  line  drawings. 

Cell  MH:  The  medium  functional-high  physical 

cell  could  contain  a partially  disabled  actual 
device.  In  this  case,  the  device  would  look 
exactly  like  the  actual  equipment,  but  would  be 
only  marginally  functional.  Cell  MM:  A medium 

functional-medium  physical  device  could  be  both 
partiall"  functional  and  also  be  degraded  In 
terms  of  size,  number  and  accuracy  of  components 
as  In  cell  HM.  Cell  ML:  The  medium  functional- 

low  physical  cell  could  contain  a device  which 
consists  of  a two-dimentlonal  display  (line 
drawings)  of  the  actual  equipment.  These  draw- 
ings would  afford  the  trainee  a means  for  Indi- 
cating control  choices  or  test  points,  but  would 
not  provide  complete  system  responses  to  these 
choices . Cell  LH:  The  device  in  the  low  func- 

tional- high  physical  cell  could  be  a totally 
disabled  piece  of  actual  equipment.  Controls 
would  be  frozen,  displays  and  test  points  would 
be  non  functional.  The  device  would  look  Just 
like  the  actual  equipment  but  would  not  work  at 
all.  Cell  1M:  The  low  functional-medium  physi- 

cal device  could  be  a totally  disabled  version  of 
the  degraded  device  used  in  cell  MH.  Cell  LL: 
Finally,  the  low  functional-low  physical  cell 
could  contain  a device  which  consists  of  a set 
of  line  drawings.  These  drawings  would  be  phy- 
sically the  same  as  those  In  cells  HL  and  ML,  but 
would  be  totally  non  functional.  These  examples 
are  not  the  only  way  the  nine  cell  matrix  could 
be  filled,  but  they  are  one  concetualization 
which  ARI  hopes  will  be  fillowed  by  others.  The 
Idea  is  to  apply  as  many  alternate  approaches 
to  the  question  of  physical  and  functional  device 
characteristics  and  transfer  of  training  as  poss- 
ible, but  always  trying  to  maintain  the  nine 
cell  matrix  as  an  organizational  framework. 

The  goal  of  the  empirical  efforts  In  this 
research  program  is  to  provide  data  on  the  rela- 
tionship of  training  device  characteristics 
(fidelity)  to  training  effectiveness  as  that 
relationship  Is  modlflec  by  the  many  training 
system  variables  which  Interact  with  fidelity. 
Previous  research  efforts  has  not  attempted  to 
systematically  control  these  Interactive  vari- 
ables but  hav  rather  looked  at  whole  devices  In 
the  context  of  already  established  training  pro- 
grams. ARI  believes  that  it  Is  only  by  conduct- 
ing controled,  systematic  experiments  will  we  be 
able  to  generate  the  necessary  data  to  provide 
useful  guidance  to  Individuals  who  must  specify 
the  characteristics  of  training  devices  and 
programs  of  Instruction. 


Table  1 displays  a list  of  some  of  the  var- 
iables which  are  believed  to  be  those  which 
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Variables  Which  Interact.  With  fidelity 


1. 


Task  Type 

- Operations 

- Maintenance 

- others 


?.  Task  Difficulty 

?.  Specific  Skills 

- Motor 

- Perceptual 

- Cognitive 

- Others 


Trainee  Poohisticatlon 

- t'ovlce 

- intermediate  ?. 

- Expert 


Stage  of  Training 

- Introduction 

- Procedural 

Training 

- familiarization 

Training 

- Transition 

Training 
Training  Context. 

- Institutional 

- Field 

Incorporation  of 
Device  Into  POI 

User  Acceptance 

- Instructors 

- Students 

Use  of  Instruc- 
tional features 


Interact  with  fidelity.  ART'S  goal  is  to 
accumulate  data  on  all  of  these  interactions 
(including  data  from  previous  experiments  and 
experiments  conducted  by  other  research 
organizations)  Into  a data  base  which  can  then 
serve  as  the  basis  for  user  oriented  guidance  In 
specifying  the  characteristics  of  a training 
simulator.  The  first  of  API's  empirical  efforts 
examined  how  physical  and  functional  fidelity 
were  related  to  training  effectiveness  In  a 
perceptual  motor  task. 

Simulator  fidelity  in  a Perceptual  Motor  Task 

Honeywell  SRC,  under  contract  to  ART, 
conducted  an  experiment  to  determine  (1) 

adequacy  of  API's  definition  of  simulator 
fidelity,  (?)  the  appropriateness  of  a nine  cell 
physical-functional  fidelity  matrix,  and  (?)  the 
relationship  of  simulator  fidelity  to  training 
effectiveness  in  a perceptual  motor  task.  Paum 
et  al.  * discussed  the  criteria  used  to 
determine  the  task  selected  in  this  experiment. 

1.  The  task  must  embody  the  skills 

required  in  an  actual  maintenance  task 

envlroment. 

?.  Task  performance  must  lend  itself  to 

straight  forward  measurement;  the  measuremetns 
must  be  valid,  reliable,  and  sensitive. 

?.  The  task  must  be  learnable  in  a 

reasonable  period  of  time. 
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Based  on  those  criteria  the  task  chosen  for 
this  experiment  was  the  truing  of  a bicycle 
wheel. 

the  wheel  truing  experiment  used  the 
fidelity  definition  and  ° cell  matrix  discused 
above.  In  this  effort,  only  *>  of  the  nine  cells 
were  investigated.  Figure  ? shows  the  nine  cell 
matrix  with  the  relevant  cells  Indicated.  By 
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Figure  2:  Application  of  the  nine  cell  matrix  in  the 

perceptual/Botor  experleent 

comparing  the  results  in  cells  HH,  MM,  and  LL  the 
general  relationship  of  fidelity  (both  physical 
and  functional)  to  training  effectiveness  was 
determined.  By  comparing  cells  HH,  KL,  LH,  and 
IX  an  indication  of  the  relative  contributions  of 
the  physical  and  functional  aspects  of  fidelity 
was  determined. 

The  data  collection  portion  of  the  wheel 
truing  experiment  was  completed  in  June  of  1982. 
Detailed  analyses  of  the  data  are  not  available 
at  the  time  of  this  writing,  but  tentative  results 
can  be  presented.  Overall,  there  was  an  improve- 
ment in  trainee  performance  under  all  training 
conditions.  Ten  t-tests  were  computed  which 
compared  the  starting  and  finishing  points  in 
each  performance  trial.  Performance  showed 
significant  Improvement  (p  less  than  .005)  in 
all  conditions. 

In  one  analysis,  the  combined  effects  of 
physical  and  functional  similarity  on  training 
effectiveness  were  assessed.  A one  way  analysis 
of  variance  (ANOVA)  was  used  to  compare  the 
performance,  on  actual  equipment,  of  subjects 
trained  on  the  devices  in  cells  HH,  Hi,  and  LL 
of  Figure  2.  In  this  analysis,  subjects  trained 


in  the  lower  fidelity  conditions  performed 
almost  as  well  as  subjects  trained  in  the  high 
fidelity  condition.  There  waa  in  fact  no 
statistically  significant  difference  between 
these  groups.  The  indication  la  therefore  that 
training  devices  for  a simple  perceptual  motor 
task  may  not  necessarily  need  to  be  designed 
with  high  fidelity. 

Another  analysis  attempts  to  separate  the 
effects  of  the  physical  and  the  functional 
aspects  of  fidelity.  This  analysis  compares 
the  performance  of  subjects  trained  on  devices 
in  cells  HH,  HL,  LH,  and  LL  of  Figure  2.  When 
these  data  are  analyzed  using  a 2x2  factorial 
design,  there  la  no  significant  effect  of  func- 
tional similarity,  but  there  is  a significant 
effect  of  physical  similarity  (F*4.157;  df»l,75; 
p less  than  .05).  In  other  words,  no  matter 
how  the  simulator  functions,  in  this  task  sub- 
jects perform  better  if  the  simulator  is  more 
physically  similar  to  the  actual  equipment. 


These  results  must  be  considered  inconclu- 
sive and  further  analyses  are  reouired.  Though 
the  above  analyses  are  only  on  terminal  perform- 
ance, additional  analyses  will  compare  the 
subject's  rate  of  learning  hv  examining  perform- 
ance over  time.  Also  some  form  of  blocking  of 
subjects  may  be  attempted  to  reduce  within  group 
variance.  Hven  so,  these  data  are  an  important 
first  step  in  validating  the  9 cell  accroach  to 
fidelity  research  and  show  that  this  is  a viable 
method  "or  generating  basic  data  on  the  rela- 
tionship of  simulator  fideltiv  to  training 
effectiveness. 

Generating  hate  on  interactive  Variables 

The  above  perceptual-motor  experiment 
yielded  valuable  data  on  the  relationship  of  the 
physical  and  functional  aspects  of  fidelity  to 
trainim  effectiveness,  but  only  for  that 
specific  task.  Other  tasks  and  other  inter- 
active variables  reoulre  further  investiga- 
tion. AFT,  through  its  basic  research  (6.1) 
program,  is  beginning  to  collect,  data  on  these 
interactive  variables. 

One  6.1  effort,  currently  underway  at  Oorge 
Mason  University,  involves  constructing  both  * 
generic  device  to  use  as  an  electro-mechanlcsl/ 
hydraulic  reference  system  and  several  degraded 
simulations  of  that  reference  system.  The  idea 
here  is  that  the  reference  system  will  serve  in 
the  role  of  actual  eoulpment.  Tt  will  function 
in  a variety  of  ways  such  as  turning  on  pumps, 
generating  tones,  turning  on  lights  or  fans, 
none  of  which  reallv  "do"  anything.  However 
malfunctions  will  he  introduced  into  the  system 
and  trainees  will  have  to  troubleshoot  snd 
repair  it. 
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Different  groups  will  be  trained  on  either 
the  actual  equipment  (the  reference  device)  or 
on  one  of  several  simulators  of  the  reference 
system  with  varied  degrees  of  fidelity.  The 
performance  of  individuals  from  each  group  will 
then  be  measured  on  the  reference  device  to  de- 
termine the  degree  of  training  transfer  for  each 
level  of  fidelity  and  mixture  of  interactive 
variables . 

This  experimental  paradigm  will  provide  an 
opportunity  to  generate,  under  controlled  lab- 
oratory conditions,  the  basic  data  needed  on 
many  of  the  interactive  variables  in  the  fidelity 
training  effectiveness  relationship  (see  Table  1). 
Data  collection  will  begin  in  the  Spring  of  1983 
after  the  reference  device  and  simulators  have 
been  constructed.  Systematic  inclusion  of  inter- 
active variables  will  begin  after  initial  base- 
line data  have  been  collected.  The  entire  effort 
is  expected  to  conclude  in  the  Sunnier  of  1985. 


The  Future:  Building  a Data  Base 

The  goal  of  ARI's  Training  Simulation  re- 
search program  is  to  produce  a user  oriented 
guidance  package  to  help  determine  the  character- 
istics which  should  be  incorporated  into  training 
simulators  to  insure  both  training  and  cost 
effectiveness.  It  is  the  strategy  of  this  re- 
search program  to  base  such  a guidance  package 
on  empirical  data . 

Data  Base  Sources 

Three  major  sources  of  data  will  be  used  to 
construct  the  training  device  characteristics 
data  base.  First,  previously  conducted  studies 
will  be  evaluated  to  determine  whether  the  data 
generated  from  these  efforts  are  suitable  for 
Inclusion  in  the  data  base.  If  so,  the  data  will 
be  categorized  and  added  to  the  data  base.  If 
not,  the  particular  study  will  be  categorized  so 
that  additional  efforts  can  fill  in  the  gap  in 
the  data  base.  At  present,  this  evaluation  of 
previous  research  is  under  way  and  it  is  expected 
to  be  completed  during  the  Fall  of  1982.  An  ARI 
Technical  Report  which  accumul- 

ates and  review  s the  literature  on  training  de- 
vice research  issues  and  which  will  serve  as  the 
organizing  framework  for  the  data  base  is  in  pre- 
paration. This  paper  is  expected  to  be  completed 
by  January,  1983. 

A second  source  of  data  for  the  data  base 
will  come  from  ARI's  research  efforts  as  des- 
cribed above.  These  efforts  will  not  attempt  to 
reinvent  the  wheel  by  duplicating  previous 
studies  which  have  been  deemed  adequate.  They 
will  rather  focus  on  gaps  in  existing  data, 
either  because  previous  efforts  have  not  add- 
ressed the  right  questions,  because  data  are  not 
valid  and  reliable,  or  because  the  results  are 
not  systematic  enough  for  inclusion  in  the  data 
base.  The  wheel  truing  experiment  described 
above  is  the  first  entry  from  this  data  source. 
Other  basic  (6.1)  and  applied  (6.2)  research 


efforts  are  presently  being  conceived  and  state- 
ments of  work  (SCWs)  are  in  preparation. 

It  is  anticipated  that  enough  data  will  be 
accumulated  by  1984  that  a preliminary  iteration 
of  the  guidance  package  may  be  produced.  This 
guidance  package  will  probably  start  as  a work- 
book but  will  eventually  be  automated  for  access 
via  computer  terminals. 


Hc^e.-sing  the  Data  Base 

Tn  order  to  make  the  training  device 
characteristics  data  base  more  than  Just  an 
academic  exercise,  it  is  necessary  to  insure 
that  the  individuals  who  need  guidance  in 
specifying  training  device  characteristics  have 
access  to  the  data  base.  At  this  time  there  are 
tentative  plans  to  automate  the  data  base  in 
order  to  allow  users  direct  access  to  the 
data.  A preliminary  step  to  automating  the  data 
base  is  to  organize  it  arround  user  oriented 
research  Issues,  This  organizational  effort  is 
currently  underway  and  a data  base  framework 
should  be  produced  in  the  Fall  of  198?. 

Once  a basic  framework  for  the  data  base  is 
developed  it  can  be  used  as  the  basis  for  an 
automated  delivery  svstem.  Such  a svstem,  using 
computer  terminals,  would  afford  access  for 
decision  makers  at  various  points  in  the  ISD 
process.  Tt  would  also  allow  freauent  updating 
as  new  and  better  data  are  developed.  the 
iterative  nature  of  the  proposed  data  base  can 
insure  that  the  training  community  will  have  the 
most  recent  data  available  on  a wide  variety  of 
training  device  research  issues.  With  access  to 
such  data,  the  development  of  training  devices 
need  not  rely  on  intuition  but  rather  or  the 
best.  available  information  on  training 
effectiveness. 

Conclusion 

APT  is  pursuing  a broad  research  program 
aimed  at.  building  a data  base  which  will  relate 
training  simulation  design  characteristics  to 
transfer  of  training  effectiveness.  An  initial 
framework  for  data  collection  built  around  a 
parsimorious  definition  of  simulation  fidelity 
has  been  constructed  and  is  being  used  to  guide 
a series  of  on-going  and  planned  laboratory 
experiments.  The  long  range  objective  is  to 
create  a comprehensive  simulation  design  data 
base  and  to  evolve  that  data  base  into  a tool 
which  can  aid  in  specifying  training  device 
recul rements. 
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